Why is uploading to S3 so slow?
- by Tom Marthenal
I am using s3cmd to upload to S3:
# s3cmd put 1gb.bin s3://my-bucket/1gb.bin
1gb.bin -> s3://my-bucket/1gb.bin [1 of 1]
366706688 of 1073741824 34% in 371s 963.22 kB/s
I am uploading from Linode, which has an outgoing bandwidth cap of 50 Mb/s according to support (roughly 6 MB/s).
Why am I getting such slow upload speeds to S3, and how can I improve them?
Update:
Uploading the same file via SCP to an m1.medium EC2 instance (SCP from my Linode to the instance's EBS drive) gives about 44 Mb/s according to iftop (any compression done by the cipher is not a factor).
Traceroute:
Here's a traceroute to the server it's uploading to (according to tcpdump).
# traceroute s3-1-w.amazonaws.com.
traceroute to s3-1-w.amazonaws.com. (72.21.194.32), 30 hops max, 60 byte packets
1 207.99.1.13 (207.99.1.13) 0.635 ms 0.743 ms 0.723 ms
2 207.99.53.41 (207.99.53.41) 0.683 ms 0.865 ms 0.915 ms
3 vlan801.tbr1.mmu.nac.net (209.123.10.9) 0.397 ms 0.541 ms 0.527 ms
4 0.e1-1.tbr1.tl9.nac.net (209.123.10.102) 1.400 ms 1.481 ms 1.508 ms
5 0.gi-0-0-0.pr1.tl9.nac.net (209.123.11.62) 1.602 ms 1.677 ms 1.699 ms
6 equinix02-iad2.amazon.com (206.223.115.35) 9.393 ms 8.925 ms 8.900 ms
7 72.21.220.41 (72.21.220.41) 32.610 ms 9.812 ms 9.789 ms
8 72.21.222.141 (72.21.222.141) 9.519 ms 9.439 ms 9.443 ms
9 72.21.218.3 (72.21.218.3) 10.245 ms 10.202 ms 10.154 ms
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
The latency looks reasonable, at least until the server stopped responding to ping requests.
